Information processing device, information processing method, program, and recording/reproduction system

ABSTRACT

To reduce uncomfortableness in the transmission of sensory stimulation information, the uncomfortableness being caused by different recording and reproducing environments. An information processing device according to the present technique includes: a positional-relationship-information acquisition unit that acquires information on a first positional relationship between a recording device and a recording subject, the recording device performing recording in a recording environment for recording sensory stimulation information transmitted from the recording subject, and information on a second positional relationship between the recording subject and a user in the reproduction environment of the recorded sensory stimulation information; and a correction unit that corrects the output signals of reproducing devices for the reproduction of the sensory stimulation information on the basis of the information on the first positional relationship and the information on the second positional relationship.

TECHNICAL FIELD

The present technique relates to an information processing device, aninformation processing method, a program, and a recording/reproductionsystem.

BACKGROUND ART

Conventionally, various techniques for presenting recorded informationabout sensory stimuli including a tactile stimulus to a user areproposed as described in, for example, PTL 1.

CITATION LIST Patent Literature

[PTL 1]

WO 2019/013056

SUMMARY Technical Problem

In recorded contents such as sports and movies, sensory stimulationinformation includes videos, sounds, and vibrations and is reproduced bya reproducing unit, so that a user can enjoy the contents even if theuser is located at a site different from a recording site.

However, such sensory stimulation information is conveyed in variousmanners with different transmission rates or damping factors dependingupon the environment. Thus, if the contents are recorded and reproducedin different environments, sound and vibrations of video may propagatein a different manner from the recognition of a user, so that the userof the contents may feel uncomfortable and the realism of the contentsmay be lost.

The present technique has been devised in view of the circumstances. Anobject of the present technique is to reduce uncomfortableness in thetransmission of sensory stimulation information, the uncomfortablenessbeing caused by different recording and reproducing environments.

Solution to Problem

An information processing device according to the present technique isan information processing device including: apositional-relationship-information acquisition unit that acquiresinformation on a first positional relationship between a recordingdevice and a recording subject, the recording device performingrecording in a recording environment for recording sensory stimulationinformation transmitted from the recording subject, and information on asecond positional relationship between the recording subject and a userin a reproduction environment of the recorded sensory stimulationinformation, and a correction unit that corrects the output signals ofreproducing devices for the reproduction of the sensory stimulationinformation on the basis of the information on the first positionalrelationship and the information on the second positional relationship.

The sensory stimulation information is various kinds of stimulationinformation about, for example, videos, sounds, sensory stimuli, andodors that are perceivable by the user.

The recording subject is assumed to be, for example, an object thatserves as a vibration source or a sound source in the recordingenvironment. The recording device is, for example, an imaging device forimaging, a microphone for collecting acoustic information, and an IMU(Inertial Measurement Unit) for collecting vibration information.

The position of the recording subject in the reproduction environment isposition information in which an object is supposed to be recognized bythe user having received the sensory stimulation information. Forexample, to the user who watches a display part (e.g., a screen or adisplay) showing a video of a basketball as a recording subject, theposition of the recording subject is the position of the basketballvirtually located through the display. Thus, the positional relationshipbetween the recording subject and the user in the reproductionenvironment is assumed to be the relationship between the virtualposition of the basketball and an actual position of the user.

The audio signal of a speaker serving as the reproducing device and theoutput signal of a tactile-signal presenting device are corrected on thebasis of the information on the first positional relationship in therecording environment and the information on the second positionalrelationship in the reproduction environment. This allows the user toperceive sensory stimulation according to the viewing position.Specifically, the closer to the display part, the higher the level ofsounds and vibrations perceivable by the user.

In the information processing device according to the present technique,if the reproducing devices include multiple kinds of reproducing devicesfor multiple kinds of sensory stimulation information, the informationon the second positional relationship may be information on therelationship between a virtual position of the recording subject, whichis perceived by the user through the display part showing a reproducedimage, and the position of the user.

In other words, the output signals of auditory information and tactileinformation are corrected according to visual information received bythe user.

In the information processing device according to the present technique,the correction unit may make the correction according to a change of thefirst positional relationship in the recording environment.

Thus, the output signal is corrected in response to a movement of therecording device or a movement of the recording subject in the recordingenvironment.

In the information processing device according to the present technique,the first positional relationship may be changed according to a changeof the position of the recording subject in the recording environment.

Thus, the output signal of the reproducing device is corrected accordingto a change of the position of the recording subject.

In the information processing device according to the present technique,the first positional relationship may be changed according to a changeof the position of the recording device in the recording environment.

Thus, the output signal of the reproducing device is corrected accordingto a change of the position of the recording device.

In the information processing device according to the present technique,the correction unit may make the correction according to a change of thesecond positional relationship in the reproduction environment.

Thus, the output signal is corrected in response to a movement of thereproducing device or a movement of the user in the reproductionenvironment.

In the information processing device according to the present technique,the second positional relationship may be changed according to a changeof the position of the user in the reproduction environment.

Thus, the output signal of the reproducing device is corrected accordingto a change of the viewing position of the user.

In the information processing device according to the present technique,the second positional relationship may be changed according to a changeof the position of the reproducing device in the reproductionenvironment.

Thus, the output signal of the reproducing device is corrected accordingto a change of the reproducing device.

In the information processing device according to the present technique,the correction unit may correct the output timing of the output signal.

Hence, for example, in the case of a large distance between thereproducing device (a display part, e.g., a display or a screen), whichreceives an outputted video, and the user, the output signal of aspeaker serving as the reproducing device can be outputted later.

In the information processing device according to the present technique,the correction unit may correct the signal strength of the outputsignal.

Hence, for example, in the case of a large distance between thereproducing device (a display part, e.g., a display or a screen), whichreceives an outputted video, and the user, the output signal of aspeaker serving as the reproducing device can be reduced in signalstrength.

In the information processing device according to the present technique,the correction unit may make the correction according to environmentinformation in the recording environment.

Thus, the output signal is corrected such that the user actually feelsthe same sensory stimulation as in the recording environment.

In the image processing device according to the present technique, theenvironment information may be air temperature information.

Thus, even if a temperature of the environment where the user isactually present is different from a temperature of the recordingenvironment, the output signal is corrected such that the user in thereproduction environment can perceive sensory stimulation to beperceived by the user in the recording environment.

In the image processing device according to the present technique, theenvironment information may be medium information.

Thus, even if the medium (e.g., water or air) of the reproductionenvironment is different from the medium of the recording environment,the output signal is corrected such that the user in the reproductionenvironment can perceive sensory stimulation to be perceived by the userin the recording environment.

In the information processing device according to the present technique,the correction unit may correct the output signal of the reproducingdevice that reproduces acoustic information as the sensory stimulationinformation.

Thus, the output signal is corrected for the acoustic information thattends to arrive at different times when the recording position and thereproduction position are different from each other.

In the information processing device according to the present technique,the correction unit may correct the output signal of the reproducingdevice that reproduces vibration information as the sensory stimulationinformation.

Thus, the output signal is corrected for the vibration information thattends to arrive at different times when the recording position and thereproduction position are different from each other.

An information processing method according to the present technique isan information processing method in which the information processingdevice is caused to perform: positional relationship informationacquisition to acquire the information on the first positionalrelationship between the recording device and the recording subject, therecording device performing recording in the recording environment forrecording the sensory stimulation information transmitted from therecording subject, and the information on the second positionalrelationship between the recording subject and the user in thereproduction environment of the recorded sensory stimulationinformation, and a correction to correct the output signals of thereproducing devices for the reproduction of the sensory stimulationinformation on the basis of the information on the first positionalrelationship and the information on the second positional relationship.

The information processing method can obtain the same operation andeffects as the information processing device according to the presenttechnique.

A program according to the present technique is a program that causesthe information processing device to perform: apositional-relationship-information acquisition function of acquiringthe information on the first positional relationship between therecording device and the recording subject, the recording deviceperforming recording in the recording environment for recording thesensory stimulation information transmitted from the recording subject,and the information on the second positional relationship between therecording subject and the user in the reproduction environment of therecorded sensory stimulation information, and a correction function ofcorrecting the output signals of the reproducing devices for thereproduction of the sensory stimulation information on the basis of theinformation on the first positional relationship and the information onthe second positional relationship.

With the program according to the present technique, the informationprocessing device according to the present technique is implemented.

In a recording/reproduction system including the recording device and areproduction control device according to the present technique, therecording device includes a sensory-stimulation-information acquisitionunit that acquires the sensory stimulation information transmitted fromthe recording subject and a detection unit that detects information onthe first positional relationship between the recording device and therecording subject, and the reproduction control device includes apositional-relationship-information acquisition unit that acquires theinformation on the first positional relationship and information on thesecond positional relationship between the recording subject and theuser in the reproduction environment of the acquired sensory stimulationinformation, and a correction unit that corrects the output signal ofthe reproducing device for the reproduction of the sensory stimulationinformation on the basis of the information on the first positionalrelationship and the information on the second positional relationship.

With the recording/reproduction system according to the presenttechnique, the audio signal of a speaker serving as the reproducingdevice and the output signal of a tactile-signal presenting device arecorrected on the basis of the information on the first positionalrelationship in the recording environment and the information on thesecond positional relationship in the reproduction environment. Thisallows the user to perceive sensory stimulation according to the viewingposition.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a configuration example of a recording/reproductionsystem according to an embodiment of the present technique.

FIG. 2 illustrates a first example of a reproduction environmentaccording to the present embodiment.

FIG. 3 illustrates a second example of the reproduction environmentaccording to the present embodiment.

FIG. 4 illustrates positional relationships in a recording environmentand the reproduction environment according to the present embodiment.

FIG. 5 indicates the properties of acoustic information and vibrationinformation according to the present embodiment.

FIG. 6 illustrates a configuration example of a recording deviceaccording to the present embodiment.

FIG. 7 illustrates a configuration example of a reproducing deviceaccording to the present embodiment.

FIG. 8 illustrates positional relationships in the recording environmentand the reproduction environment according to the present embodiment.

FIG. 9 is a flowchart of correction of sensory stimulation informationaccording to the present embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment according to the present technique will bedescribed in the following order.

<1. Configuration example of recording/reproduction system><2. Difference in transmission among sensory stimulation information><3. Configuration example of recording device><4. Configuration example of reproduction control device><5. Correction of sensory stimulation information>

<6. Summary>

<7. Present technique>

As configurations illustrated in the drawings referred to in thedescription, only main configurations are extracted according to thepresent technique. The configurations illustrated in the drawings can bechanged in various ways according to the design without departing fromthe technical spirit of the present technique.

Hereinafter, the previously described configurations may be referred towith the same reference numerals and the description thereof may beomitted.

Furthermore, the present technique is not limited to the presentembodiment and includes modifications and changes as long as the objectof the present technique can be attained.

1. Configuration Example of Recording/Reproduction System

Referring to FIG. 1 , a configuration example of arecording/reproduction system 1 will be described below. Therecording/reproduction system 1 includes a recording device 2, areproduction control device 3, and reproducing devices 4.

An environment for implementing the recording/reproduction system 1 isdivided into a recording environment and a reproduction environment.

The recording device 2 records sensory stimulation informationtransmitted from a recording subject in the recording environment.

The recording subject is assumed to be, for example, an object or aperson that serves as a vibration source or a sound source in therecording environment.

The recording device 2 includes, for example, an imaging device forimaging, a microphone for collecting acoustic information, and an IMU(Inertial Measurement Unit) for collecting vibration information.

The sensory stimulation information is various kinds of stimulationinformation about, for example, videos, sounds, sensory stimuli, andodors that are perceivable by a user.

The recording device 2 generate encoded data Ec by encoding collectedsensory stimulation information.

The reproduction control device 3 performs reproduction control forallowing the user to perceive various kinds of sensory stimulationinformation recorded in the recording environment. For the reproductioncontrol, the reproduction control device 3 acquires the encoded data Ecthat has been encoded in the recording environment.

The reproduction control device 3 may acquire the encoded data Ec byoptionally provided recording media such as a magnetic disk, an opticaldisk, a magneto-optical disk, or a semiconductor memory.

The reproduction control device 3 outputs sensory stimulationinformation, which is obtained by decoding the acquired encoded data Ec,to the reproducing devices 4.

The sensory stimulation information is reproduced by each of thereproducing devices 4, allowing the user in the reproduction environmentto perceive the sensory stimulation information recorded in therecording environment.

Referring to FIG. 2 , a configuration example of the reproducing device4 will be described below.

FIG. 2 illustrates an example in which a user Hm on a chair 5 in thereproduction environment perceives the sensory stimulation informationoutputted from the reproducing device 4.

The reproducing device 4 is provided as multiple kinds of devicesaccording to multiple kinds of sensory stimulation information. Forexample, a video reproducing device 4 a, a sound reproducing device 4 b,and a vibration reproducing device 4 c are provided as the reproducingdevices 4.

The video reproducing device 4 a is a device for outputting video thatstimulates the vision of the user Hm. For example, the video reproducingdevice 4 a is a TV (television) device having a display part asillustrated in FIG. 2 . The video reproducing device 4 a may be appliedto other examples as a projector and a screen in a movie theater or thelike and a head mounted display (HMD) on the user Hm.

The sound reproducing device 4 b is a device for outputting sound thatstimulates the hearing sense of the user Hm. For example, the soundreproducing device 4 b is provided as headphones or earphones asillustrated in FIG. 2 . The sound reproducing device 4 b may be appliedto other examples as a device wearable by the user Hm and speakers in amovie theater or a room.

The vibration reproducing device 4 c is a device for outputtingvibrations that stimulate the tactile sense of the user Hm. For example,the vibration reproducing device 4 c is provided as a vibration vesthaving a vibrator or an actuator as illustrated in FIG. 2 .

The vibration reproducing device 4 c may be applied to other deviceshaving vibration devices. For example, the vibration reproducing device4 c may be applied to smart devices such as a smart phone, a controllerused for a game or an attraction, or a chair for the user Hm. Moreover,the vibration reproducing device 4 c may be applied to the walls andceiling of an attraction ride and a floor including a vibration device.

The configuration illustrated in FIG. 2 is merely exemplary, and theconfiguration example of the reproducing device 4 is not limited to thisconfiguration.

The video reproducing device 4 a, the sound reproducing device 4 b, andthe vibration reproducing device 4 c were described as the reproducingdevices 4. For example, some or all of the reproducing devices 4 may beintegrated. A chair 5 for the user Hm may act as, for example, the soundreproducing device 4 b and the vibration reproducing device 4 c thathave a sound reproducing unit and a vibration reproducing unit.Alternatively, as illustrated in FIG. 3 , a head mounted display HMD mayact as the video reproducing device 4 a and the sound reproducing device4 b.

In other cases, the reproducing devices 4 of the same type may beprovided. For example, as the vibration reproducing devices 4 c, avibration vest on the upper body of the user Hm and a vibrationsupporter on a knee of the user Hm may be provided.

With this configuration, the recording/reproduction system 1 accordingto the present embodiment is implemented.

2. Difference in Transmission Among Sensory Stimulation Information

Referring to FIGS. 4 and 5 , problems to be solved by the presentembodiment will be described below. FIG. 4 schematically illustrates thepositional relationship between a recording subject and the recordingdevice 2 in the recording environment and the positional relationshipbetween the recording subject and the user Hm in the reproductionenvironment.

In FIG. 4 , a subject position T indicates the position of the recordingsubject, a recording position R indicates the position of the recordingdevice 2 in the recording environment, a perception position A indicatesthe position of the user Hm in the reproduction environment, and aperception position B indicates the position of another user Hm. Theperception position A and the perception position B as the positions ofthe user Hm are assumed to be the reproduction positions of the soundreproducing device 4 b and the vibration reproducing device 4 c.

The subject position T in the reproduction environment is a positionsupposed to have an object to be recognized by the user Hm havingreceived the sensory stimulation information. For example, the subjectposition T is a virtual position of a recording subject to be recognizedby the user Hm through the display part of the video reproducing device4 a that reproduces video (image).

Specifically, to the user Hm who watches a display (the display part ofthe video reproducing device 4 a) showing a video of, for example, abasketball as a recording subject, the subject position T is theposition of the basketball virtually located through the display. Thus,the positional relationship between the recording subject and the userHm in the reproduction environment is assumed to be the relationshipbetween the virtual position of the basketball and an actual position(watching position) of the user Hm.

In the present embodiment, it is assumed as an example that the user Hmwearing headphones (sound reproducing device 4 b) and a vibration vest(vibration reproducing device 4 c) watches video displayed on thedisplay (video reproducing device 4 a) as illustrated in FIG. 2 .

To allow the user Hm at the perception position A to perceive, withoutlosing realism, the sensory stimulation information recorded by therecording device 2, the subject position T viewed from the recordingposition R in the recording environment is preferably located at thesubject position T (that is, the virtual position of the recordingsubject) viewed from the perception position A in the reproductionenvironment.

However, in most cases, the positional relationship between the subjectposition T and the recording position R in the recording environment andthe positional relationship between the subject position T and theperception position A in the reproduction environment are actuallydifferent from each other.

Specifically, in the example of FIG. 4 , the perception position A andthe perception position B in the reproduction environment are locatedmore remotely from the subject position T than the recording position Rin the recording environment. The perception position B is located moreremotely from the subject position T than the perception position A.

In this case, Table 1 of FIG. 4 indicates the arrival times of thesensory stimulation information to the positions from the subjectposition T. In Table 1, Time T0 indicates a time when the sensorystimulation information is transmitted from the subject position T. Atime passes in the rightward direction along the horizontal axis.

Moreover, in Table 1 of FIG. 4 , “Tr” indicates a time when the sensorystimulation information from the subject position T reaches therecording position R, “Ta” indicates a time when the sensory stimulationinformation reaches the perception position A, and “Tb” indicates a timewhen the sensory stimulation information reaches the perception positionB.

According to Table 1, video information of the sensory stimulationinformation has a transmission rate based on a speed of light and thusis hardly affected by the positional relationship from the subjectposition T, and the times of transmission to the positions (recordingposition R, perception position A, perception position B) aresubstantially equal to one another.

In contrast, acoustic information and vibration information haveconsiderably lower transmission rates than the video information andthus are susceptible to a distance from the subject position T asindicated in FIG. 4 , and the times of transmission to the positionsincrease with a distance from the subject position T.

As indicated in FIG. 5 , the acoustic information and the vibrationinformation are considerably different from each other in transmissionrate (Table 1), transmission distance for a certain time (Table 2), andarrival time at a predetermined distance (Table 3). Specifically, thevibration information has a higher transmission rate and a shorterarrival time at a predetermined distance than the acoustic information.

The difference is made by a difference between media used fortransmission. In Tables 1 to 3 of FIG. 5 , for example, air is used fortransmitting the acoustic information (sound) while solids such as ironand wood are used for transmitting signal information.

Thus, if the sensory stimulation information recorded by the recordingdevice 2 at a time corresponding to the positional relationship(direction and distance) between the subject position T and therecording position R is outputted as it is to the perception position Aor the perception position B that has a different positionalrelationship with the subject position T, transmissions expected by theuser Hm may deviate from the transmissions of the video information, theacoustic information, and the vibration information, so that the user Hmmay feel uncomfortable.

Specifically, the acoustic information recorded in the recordingenvironment is recorded after the video information because of adifference in transmission rate. The amount of delay is supposed tocorrespond to a distance between the recording subject and the recordingposition R.

In the reproduction environment, the amount of delay of the acousticinformation desirably corresponds to the positional relationship betweenthe virtual position (subject position T) of the recording subjectperceived by the user Hm and the perception position A of the user Hm.

Thus, if a distance from the subject position T to the recordingposition R in the recording environment is different from a distancefrom the subject position T, which serves as a virtual position, to theperception position A in the reproduction environment, the acousticinformation to be perceived at the recording position R is perceived bythe user Hm at the perception position A by reproducing, without anycorrections, the acoustic information recorded at the recording positionR. This may cause the user Hm to feel uncomfortable.

For example, if a distance between the recording subject and therecording position R in the recording environment is larger (greater)than a distance between the position of the user Hm and the virtualposition of the recording subject in the reproduction environment, theacoustic information reaches the user Hm earlier than expected, so thatthe realism is lost.

Thus, in the present embodiment, if the positional relationship betweenthe subject position T and the recording position R in the recordingenvironment and the positional relationship between the subject positionT (the virtual position) and the perception position (the perceptionposition A or the perception position B) in the reproduction environmentare different from each other, the acoustic information outputted fromthe sound reproducing device 4 b and the vibration information outputtedfrom the vibration reproducing device 4 c are corrected. The correctionof the acoustic information and the vibration information can suppress adeviation of sounds and vibrations, which are perceived by the user Hm,from expectations, thereby preventing a loss of realism.

3. Configuration Example of Recording Device

Referring to FIG. 6 , a configuration example of the recording device 2according to the present embodiment will be described below.

The recording device 2 is configured with, for example, computer devicessuch as a CPU (Central Processing Unit) and a DSP (Digital SignalProcessor). Various programs are executed by a CPU or a DSP and thelike, so that the recording device 2 is provided with various functions.

The recording device 2 includes a camera unit 10, an image signalprocessing unit 11, a sound input unit 12, a sound signal processingunit 13, a vibration detecting unit 14, a vibration signal processingunit 15, a sensor unit 16, an encoding unit 17, a storage unit 18, and acommunication unit 19. The image signal processing unit 11, the soundsignal processing unit 13, the vibration signal processing unit 15, thesensor unit 16, the encoding unit 17, the storage unit 18, and thecommunication unit 19 are connected to one another via a bus 20 andexchange various kinds of data and a control signal or the like.

The camera unit 10 is provided with an imaging unit including an imagingdevice, which is configured with, for example, a CCD (Charged-coupleddevices) image sensor or a CMOS (Complementarymetal-oxide-semiconductor) image sensor, and an imaging optical system.The imaging unit receives light based on a subject (recording subject)imaged by a lens system and converts the light into an electric signal(captured image signal). The camera unit 10 supplies the convertedcaptured image signal to the image signal processing unit 11.

The image signal processing unit 11 is configured as, for example, animage processor like a DSP (Digital Signal Processor) and obtains imagedata (captured image data) by performing various kinds of signalprocessing including A/D (Analog/Digital) conversion on the capturedimage signal from the camera unit 10. The captured image data obtainedby the image signal processing unit 11 is supplied to the encoding unit17.

The captured image data may be temporarily stored (recorded) in thestorage unit 18.

The sound input unit 12 is configured with, for example, a microphone.The sound input unit 12 collects a sound from the recording subject andconverts the sound into an electric signal (sound signal). The soundinput unit 12 supplies the converted electric signal to the sound signalprocessing unit 13.

The sound signal processing unit 13 obtains sound data by performingvarious kinds of signal processing including amplification and A/Dconversion on the sound signal. The sound data obtained by the soundsignal processing unit 13 is supplied to the encoding unit 17.

The sound data may be temporarily stored in the storage unit 18.

The vibration detecting unit 14 is, for example, a vibration-detectinggyro sensor for an IMU or the like. The vibration detecting unit 14detects vibrations from the recording subject serving as a source andconverts the vibrations into an electric signal (vibration signal). Thevibration detecting unit 14 supplies the converted electric signal tothe vibration signal processing unit 15.

The vibration signal processing unit 15 obtains vibration data (tactiledata) by performing various kinds of signal processing includingamplification and A/D conversion on the vibration signal. The vibrationdata obtained by the vibration signal processing unit 15 is supplied tothe encoding unit 17.

The vibration data may be temporarily stored in the storage unit 18.

The vibration data may be generated by filtering the sound data, orproduced and edited vibration data may be recorded in advance in thestorage unit 18.

The sensor unit 16 includes a position sensor for detecting the positionof the recording device 2, a range sensor for detecting a distance fromthe recording device 2 to the recording subject, a temperature sensorfor detecting a temperature of the recording environment, and varioussensors for detecting the media of the recording environment, and thesensor unit 16 is configured to detect various kinds of information.

The camera unit 10 can also act as a range sensor. For example, thecamera unit 10 includes two imaging units “laterally separated from eachother. The imaging units laterally separated from each other cangenerate a parallax between captured images so as to obtain distanceinformation. Specifically, information on a distance to the recordingsubject can be obtained according to a stereo method on the basis ofimages captured by the imaging units. In other words, three-dimensionaldata can be obtained so as to indicate positions to the recordingsubject in the directions of the X axis, the Y axis, and the Z axis in areal space. Alternatively, the camera unit 10 including a TOF(Time-of-Flight) sensor or an image-surface phase difference sensor mayact as a range sensor.

The encoding unit 17 encodes the sensory stimulation information, whichis supplied from the image signal processing unit 11, the sound signalprocessing unit 13, and the vibration signal processing unit 15,according to a predetermined data format and stores the encoded data Ec,which is obtained by the encoding, in the storage unit 18.

In the storage unit 18, the encoded data Ec obtained by the encodingunit 17 is temporarily stored. Moreover, signal data from the signalprocessing units including the image signal processing unit 11, thesound signal processing unit 13, and the vibration signal processingunit 15 and various kinds of detection information from the sensor unit16 may be temporarily stored in the storage unit 18.

The storage unit 18 may be a removable recording medium, e.g., a memorycard, an optical disk, or magnetic tape, a fixed HDD (Hard Disk Drive),or a fixed semiconductor memory module.

The communication unit 19 communicates with an external device.Specifically, the communication unit 19 includes a module for radiocommunications, so-called 3G or 4G communications via a wireless LAN(Local Area Network) according to a mobile communication system for acellular-phone system and the IEEE802.11 series, or wire communicationsvia a wired LAN or a USB. Thus, various kinds of data can be exchangedwith an external device, e.g., the reproduction control device 3.

With this configuration, the recording device 2 according to the presentembodiment is implemented.

The present embodiment described an example where the camera unit 10,the sound input unit 12, and the vibration detecting unit 14 are allprovided in the recording device 2. Some or all of the units may beprovided in an external device capable of communicating with therecording device 2. In this case, in order to acquire positioninformation on the units, a position sensor is provided for the externaldevice. The external device supplies position information on theexternal device to the recording device 2 in addition to the acquiredsensory stimulation information.

Each external device may be provided with various sensors or the likefor acquiring environment information. In this case, the environmentinformation is temperature information, humidity information, or mediuminformation and serves as a factor responsible for a change of atransmission speed of the sensory stimulation information.

Alternatively, an external device including the camera unit 10, thesound input unit 12, and the vibration detecting unit 14 may be providedas another recording device 2. In this case, for each external device,the recorded sensory stimulation information is subjected to variouskinds of signal processing including amplification and A/D conversionand is encoded, so that the encoded data Ec is generated.

4. Configuration Example of Reproduction Control Device

Referring to FIG. 7 , a configuration example of the reproductioncontrol device 3 according to the present embodiment will be describedbelow.

The reproduction control device 3 is configured with, for example,computer devices such as a CPU and a DSP. Various programs are executedby a CPU or a DSP and the like, so that the reproduction control device3 is provided with various functions.

The reproduction control device 3 includes a decoding unit 30, a signalprocessing unit 31, a reproduced data generating unit 32, a storage unit33, and a communication unit 34. The decoding unit 30, the signalprocessing unit 31, the reproduced data generating unit 32, the storageunit 33, and the communication unit 34 are connected to one another viaa bus 40 and exchange various kinds of data and a control signal or thelike.

The decoding unit 30 decodes the encoded data Ec, which has been encodedin the recording device 2, according to a predetermined data format. Bydecoding the encoded data Ec, the decoding unit 30 acquires image data,sound data, vibration data that have been obtained by recording thesensory stimulation information.

The signal processing unit 31 is configured with a microcomputerincluding a CPU, a ROM (Read Only Memory), and a RAM (Random AccessMemory).

The signal processing unit 31 has the functions of apositional-relationship-information acquisition unit 35 and a correctionunit 36.

The positional-relationship-information acquisition unit 35 acquiresinformation on a first positional relationship that is a positionalrelationship between the recording device 2 and the recording subject inthe recording environment and a second positional relationship that is apositional relationship between the recording subject (virtual position)and the user Hm in the reproduction environment of the recorded sensorystimulation information.

Referring to FIG. 8 , the first positional relationship and the secondpositional relationship will be described below. FIG. 8 conceptuallyillustrates the positional relationship among the recording subject, therecording device 2, and the user Hm. As in FIG. 4 , a subject position Tindicates the position of the recording subject, a recording position Rindicates the position of the recording device 2, and a perceptionposition A indicates the position of the user Hm.

Moreover, in this example, the video reproducing device 4 a includes ascreen serving as a display part.

In the recording environment, the recording device 2 records sensorystimulation information transmitted from recording subjects such as abasket ball and a player in a basketball court BC. In the reproductionenvironment, the user Hm wearing the sound reproducing device 4 b andthe vibration reproducing device 4 c as illustrated in FIG. 2 watchesthe output video of the video reproducing device 4 a on a screen.

In this case, the first positional relationship is the positionalrelationship between the subject position T and the recording position Rin the recording environment. For example, the first positionalrelationship can be expressed by a distance Lx between the subjectposition T and the recording position R.

The second positional relationship is the positional relationshipbetween a virtual position (subject position T) of the recording subjectperceived by the user Hm through the screen and the perception positionA in the reproduction environment. For example, the second positionalrelationship can be expressed by a distance Ly between the virtualsubject position T and the perception position A.

The distance Ly between the virtual position of the recording subjectand the perception position A is the sum of the distance Lx between thesubject position T and the recording position R and a distance Lz fromthe screen (the display part of the video reproducing device 4 a) to theperception position A.

When the user Hm wears the reproducing device 4, the position of theattached reproducing device 4 is equivalent to the perception position Aof the user Hm. Thus, the second positional relationship can be regardedas the positional relationship between the recording subject at thevirtual position and the reproducing device 4 attached to the user Hm.In this case, the positional relationship includes various positionalrelationships such as one-dimensional to three-dimensional positionalrelationships.

The positional-relationship-information acquisition unit 35 in FIG. 7acquires information on the positional relationship (first positionalrelationship) between the subject position T and the recording positionR on the basis of the acquired position information on the recordingdevice 2.

The positional-relationship-information acquisition unit 35 specifiesthe positional relationship between the perception position A and therecording position R by acquiring position information on thereproducing devices 4 of various types and acquires (calculates)information on the second positional relationship on the basis of thepositional relationship between the perception position A and therecording position R and the positional relationship (first positionalrelationship) between the recording position R and the subject positionT.

On the basis of the information on the first positional relationship andthe information on the second positional relationship, the informationbeing acquired by the positional-relationship-information acquisitionunit 35, the correction unit 36 corrects an output signal to thereproducing device 4 that reproduces the sensory stimulationinformation. For example, by correcting the output timing of the outputsignal, the correction unit 36 adjusts a time before the sensorystimulation information reaches the user Hm (perception position A).

Furthermore, by correcting the signal strength of the output signal, thecorrection unit 36 adjusts the intensity of the sensory stimulationinformation perceived by the user Hm at the perception position A.

The correction unit 36 corrects the output signal of the reproducingdevice 4 according to a change of the first positional relationship anda change of the second positional relationship.

The first positional relationship changes according to, for example, amovement of the recording subject or a movement of the recording device2 in the recording environment. The second positional relationshipchanges according to, for example, a movement of the user Hm ormovements of the reproducing devices 4 of various types in thereproduction environment. The processing for correcting the outputsignal of the reproducing device 4 will be specifically described later.Hence, the various functions of the signal processing unit 31 accordingto the present embodiment are implemented.

The reproduced data generating unit 32 performs predetermined processingcorresponding to the reproducing device 4 at the destination, forexample, codec processing or packetization on the output signalcorrected by the signal processing unit 31.

In the storage unit 33, the reproduced data that is generated by thereproduced data generating unit 32 is temporarily stored.

The storage unit 33 may be a removable recording medium, e.g., a memorycard, an optical disk, or magnetic tape, a fixed HDD (Hard Disk Drive),or a fixed semiconductor memory module.

If the storage unit 33 is a removable recording medium, the recordingmedium in which the encoded data Ec is recorded can be connected to thereproduction control device 3 so as to supply the encoded data Ec to thedecoding unit 30.

The communication unit 34 communicates with external devices like thecommunication unit 19 of FIG. 6 . This allows the communication unit 34to receive the encoded data Ec from the recording device 2.

Moreover, the communication unit 34 transmits the reproduced data, whichis generated by the reproduced data generating unit 32, to thereproducing devices 4 of various types. The communication unit 34 mayacquire position information or the like from the reproducing devices 4of various types.

Hence, the reproduction control device 3 according to the presentembodiment is implemented.

5. Correction of Sensory Stimulation Information

Referring to FIG. 9 , a correction of the sensory stimulationinformation by the signal processing unit 31 of the reproduction controldevice 3 will be described below. In the following example, the user Hmwearing the sound reproducing device 4 b and the vibration reproducingdevice 4 c as illustrated in FIG. 2 visually recognizes video displayedon the display part of the video reproducing device 4 a.

In step S101, the signal processing unit 31 first calculates a distancebetween the recording subject and the recording device 2. For example,the signal processing unit 31 may acquire information on the subjectposition T and the recording position R to calculate the distance Lx ormay acquire information on a distance from the recording device 2 to thesubject position T as the distance Lx, the information being acquired bythe recording device 2.

In step S102, the signal processing unit 31 then calculates the distanceLy between the recording subject and the user Hm. In this case, asillustrated in FIG. 8 , the distance Ly can be calculated by summing thedistance Lx and the distance Lz. Since the distance Lx is calculated bythe processing of step S101, the signal processing unit 31 can determinethe distance Ly by calculating the distance Lz.

The distance Lz can be calculated on the basis of the positionalrelationship between the user Hm and the recording position R. Since theuser Hm wears the sound reproducing device 4 b, the position of the userHm can be detected by detecting the position of the sound reproducingdevice 4 b. In the example of FIG. 2 , the position of the vibrationreproducing device 4 c attached to the user Hm may be detected.

As illustrated in FIG. 8 , the display part of the video reproducingdevice 4 a can be regarded as the recording position R. Thus, thedistance Lz between the recording position R and the user Hm can beregarded as a distance between the display part of the video reproducingdevice 4 a and the sound reproducing device 4 b.

Some methods can be used to calculate a distance between the displaypart of the video reproducing device 4 a and the sound reproducingdevice 4 b. For example, the video reproducing device 4 a may beprovided with a distance-measuring unit that calculates a distance fromthe sound reproducing device 4 b (or the user Hm). Alternatively, adistance may be calculated from position information on the display partof the video reproducing device 4 a and position information on thesound reproducing device 4 b in the same coordinate system.

If seat positions are determined in advance as in a movie theater, adistance between the seat of the user Hm and a screen may be acquiredfrom a database in which distance information on the seat positions andthe screen is stored.

In this way, the signal processing unit 31 calculates (acquires) adistance between the subject position T (virtual position) and theperception position A. The distance Ly indicates the second positionalrelationship serving as a virtual positional relationship between therecording subject and the user Hm.

In step S103, the signal processing unit 31 corrects the output signalof the sensory stimulation information to the reproducing devices 4. Thesignal processing unit 31 corrects the output timing (delay amount) andthe signal strength (attenuation) of the output signal on the basis ofthe first positional relationship and the second positionalrelationship.

For example, when a distance from the subject position T to therecording position R in FIG. 8 is denoted as “Lx[m]”, a virtual distancefrom the subject position T to the perception position A is denoted as“Ly[m]”, and a sound transmission rate is denoted as “Vaudio[m/s]”,“Taudio[mS]” representing a correction amount for a delay amount of theoutput timing of acoustic information is calculated as expressed in, forexample, [Formula 1] below.

Taudio=(Ly·Lx)/Vaudio   [Formula 1]

The calculated value of Taudio indicates a time for propagating soundfrom the recording position R to the perception position A.

When a vibration transmission rate is denoted as “Vhaptics[m/s]”,“Thaptics[mS]” representing a correction amount for a delay amount ofvibration-information output timing to be corrected is calculated asexpressed in, for example, [Formula 2] below.

Thaptics=(Ly·Lx)/Vhaptics   [Formula 2]

The calculated value of Thaptics indicates a time for propagating soundfrom the recording position R to the perception position A.

A correction amount “Gaudio[dB]” for an attenuation of the acousticinformation is calculated as expressed in [Formula 3]. “10” representsthe base of a logarithm and (Ly/Lx) represents the antilogarithm of alogarithm in [Formula 3].

Gaudio=20 log(10,(Ly/Lx))   [Formula 3]

The calculated value of Gaudio indicates an attenuation while a soundpropagates from the recording position R to the perception position A.

A correction amount “Ghaptics[dB]” for an attenuation of the vibrationinformation is calculated as expressed in [Formula 4]. “10” representsthe base of a logarithm and (Ly/Lx) represents the antilogarithm of alogarithm in [Formula 4].

Ghaptics=20 log(10,(Ly/Lx))   [Formula 4]

The calculated value of Ghaptics indicates an attenuation while a soundpropagates from the recording position R to the perception position A.

The signal processing unit 31 calculates the correction amounts of theacoustic information and the vibration information on the basis of[Formula 1] to [Formula 4].

Also for the video information, a correction amount for the delay amountof the output timing and a correction amount for an attenuation may becalculated.

However, a delay amount and an attenuation of the video information arequite smaller than a transmission system delay and thus a correctionamount for the delay amount of the output timing and a correction amountfor an attenuation are not calculated for the video information.

In the correction of step S103, the signal processing unit 31 makes acorrection according to the environment information in the recordingenvironment.

Specifically, according to the environment information, the signalprocessing unit 31 corrects a delay amount of the output timing and anattenuation in the sensory stimulation information to be outputted tothe reproducing devices 4.

For example, in the case of a correction based on an air temperature,the signal processing unit 31 acquires air temperature information onthe recording environment from the recording device 2. The signalprocessing unit 31 calculates correction amounts for a delay amount ofthe output timing and an attenuation of the acoustic information and thevibration information on the basis of the acquired air temperatureinformation.

In the correction of step S103, the signal processing unit 31 may make acorrection on the basis of air temperature information on thereproduction environment. In this case, the signal processing unit 31acquires air temperature information on the reproduction environmentfrom the reproducing devices 4 provided with temperature sensors andcalculates correction amounts for a delay amount of the output timingand an attenuation of the acoustic information and the vibrationinformation on the basis of the acquired air temperature information.

Thus, even if a temperature of the reproduction environment where theuser Hm is actually present is different from a temperature of therecording environment, the various kinds of sensory stimulationinformation are corrected to allow the user Hm in the reproductionenvironment to perceive sensory stimulation supposed to be perceived bythe user Hm if the user Hm were in the recording environment.

In the correction of step S103, the signal processing unit 31 can make acorrection in the recording environment according to a medium used fortransmitting sensory stimulation. For example, the signal processingunit 31 acquires medium information on the recording environment fromthe recording device 2.

The medium information is information for specifying, for example, amedium that transmits the acoustic information and the vibrationinformation. Specifically, the medium information is information onwater, air, or floor materials.

The signal processing unit 31 calculates correction amounts for a delayamount of the output timing and an attenuation of the acousticinformation and the vibration information on the basis of the mediuminformation, which is acquired from the recording device 2, duringrecording.

The signal processing unit 31 may make a correction on the basis of, forexample, medium information on the reproduction environment from thereproducing devices 4. At this point, the signal processing unit 31calculates correction amounts for a delay amount of the output timingand an attenuation of the acoustic information and the vibrationinformation on the basis of the medium information, which is acquiredfrom the reproducing devices 4, in the reproduction environment.

Thus, even if the medium (e.g., water or air) of the reproductionenvironment is different from the medium of the recording environment,the various kinds of sensory stimulation information are corrected toallow the user Hm in the reproduction environment to perceive sensorystimulation supposed to be perceived by the user Hm if the user Hm werein the recording environment.

In step S104, the signal processing unit 31 then sets a filter factorfor correcting a delay amount of the output timing and an attenuation ofthe sensory stimulation information, on the basis of the correctionamounts calculated in step S103. The signal processing unit 31 correctsthe various kinds of sensory stimulation information on the basis of thefilter factor.

Thus, an output signal is outputted to the reproduced data generatingunit 32 after the delay amount and the attenuation are properlycorrected on the basis of the filter factor.

In step S105, the reproduction control device 3 performs reproductioncontrol based on the corrected sensory stimulation information.

In the reproduction control, for example, reproduced data generated forthe video reproducing device 4 a by the reproduced data generating unit32 from the video information acquired from the encoded data Ec istransmitted to the video reproducing device 4 a by the communicationunit 34, enabling an output by the video reproducing device 4 a on thebasis of the video information.

Furthermore, reproduced data generated for the sound reproducing device4 b by the reproduced data generating unit 32 from the correctedacoustic information is transmitted to the sound reproducing device 4 bby the communication unit 34, enabling an output by the soundreproducing device 4 b on the basis of the corrected acousticinformation.

Moreover, reproduced data generated for the vibration reproducing device4 c by the reproduced data generating unit 32 from the correctedvibration information is transmitted to the vibration reproducing device4 c by the communication unit 34, enabling an output by the vibrationreproducing device 4 c on the basis of the corrected vibrationinformation.

If the first positional relationship or the second positionalrelationship is not changed, the signal processing unit 31 performsprocessing in the order from steps S106, S107, and S108. Thereafter,until the output of information relating to the various kinds of sensorystimulation information to the reproducing devices 4 is terminated, forexample, at the completion of the showing of the contents in step S111,the corrected acoustic information and vibration information are usedfor reproduction on the basis of the filter factor set in step S104.

If the first positional relationship or the second positionalrelationship is changed, steps S109 and S110 are performed on the basisof determination in steps S106, S107, and S108 so as to update thefilter factor.

Specifically, in step S106, the presence or absence of a change of thesubject position T of the recording subject is confirmed. If a change isconfirmed, the signal processing unit 31 advances to step S109. Inresponse to a change of the subject position T, the positionalrelationship (distance Lx) between the recording subject (subjectposition T) and the recording device 2 (recording position R) and thepositional relationship (distance Ly) between the recording subject(subject position T) and the user Hm (perception position A) arechanged. In other words, the first positional relationship and thesecond positional relationship are changed.

A change of the subject position T of the recording subject may be, forexample, a movement of the recording subject during recording in therecording environment, a change from a video of the recording subject ata remote location to a video of the recording subject at a closelocation during reproduction by the video reproducing device 4 a in thereproduction environment, or a change of video scenes (viewpoints ofrecording). Examples of a change of the subject position T of therecording subject may include a change of videos when the user Hmwearing the head mounted display HMD turns around.

In this case, in step S109, the signal processing unit 31 recalculatesthe distance Lx and the distance Ly as in step S103 and substitutes thecalculated distance Lx and distance Ly into [Formula 4] from [Formula1], thereby recalculating correction amounts for a delay amount of theinput timing and an attenuation of the sensory stimulation information.

In step S110, the signal processing unit 31 updates (resets) the filterfactor for correcting a delay amount of the output timing and anattenuation of the sensory stimulation information, on the basis of thecorrection amounts calculated in step S109. The signal processing unit31 then corrects the various kinds of sensory stimulation information onthe basis of the filter factor.

When the reproduction control is not completed, the signal processingunit 31 performs processing in the order from step S111 to step S105. Instep S105, the signal processing unit 31 performs the reproductioncontrol of the reproducing devices 4 on the basis of the sensorystimulation information that is corrected on the basis of the updatedfilter factor. Thereafter, the signal processing unit 31 performs thesame processing.

If the subject position T of the recording subject is not changed instep S106, the signal processing unit 31 confirms the presence orabsence of a change of the recording position R of the recording device2 in step S107. If a change is confirmed, the signal processing unit 31advances to step S109. This changes the positional relationship(distance Lx) between the recording subject (subject position T) and therecording device 2 (recording position R). In other words, the firstpositional relationship is changed.

A change of the recording position R may be, for example, a movement ofthe recording device 2 in the recording environment or a movement of thedisplay part of the video reproducing device 4 a that is assumed to bethe recording position R in the reproduction environment.

In this case, in step S109, the signal processing unit 31 recalculatesthe distance Lx and the distance Ly as in step S103 and recalculatescorrection amounts for a delay amount of the input timing and anattenuation of the sensory stimulation information on the basis of[Formula 1] to [Formula 4].

In step S110, the signal processing unit 31 updates (resets) the filterfactor for correcting a delay amount of the output timing and anattenuation of the sensory stimulation information, on the basis of thecorrection amounts calculated in step S109. The signal processing unit31 then corrects the various kinds of sensory stimulation information onthe basis of the filter factor.

When the reproduction control is not completed, the signal processingunit 31 performs processing in the order from step S111 to step S105. Instep S105, the signal processing unit 31 performs the reproductioncontrol of the reproducing devices 4 on the basis of the sensorystimulation information that is corrected on the basis of the updatedfilter factor.

Thereafter, the signal processing unit 31 performs the same processing.

If the recording position R of the recording device 2 is not changed instep S107, the signal processing unit 31 confirms the presence orabsence of a change of the perception position A of the user Hm in stepS108. If a change is confirmed, the signal processing unit 31 advancesto step S109.

At this point, in response to a change of the perception position A, thepositional relationship (distance Ly) between the recording subject(subject position T) and the user Hm (perception position) is changed.In other words, the second positional relationship is changed.

A change of the perception position A of the user Hm may be a movementof a vehicle carrying the user Hm or a movement of the user Hm.

The signal processing unit 31 can confirm a change of the perceptionposition A of the user Hm when the position of the reproducing device 4attached to the user Hm is changed. This is because the position of thereproducing device 4 can be regarded as the position of the user Hm. Inother words, the positional relationship (distance Ly) between therecording subject and the user Hm is changed in response to a change ofthe position of the reproducing device 4 attached to the user Hm.

In this case, when the perception position A of the user Hm changes instep S108, the signal processing unit 31 recalculates the distance Lxand the distance Ly in step S109 as in step S103 and recalculatescorrection amounts for a delay amount of the input timing and anattenuation of the sensory stimulation information on the basis of[Formula 1] to [Formula 4].

In step S110, the signal processing unit 31 updates (resets) the filterfactor for correcting a delay amount of the output timing and anattenuation of the sensory stimulation information, on the basis of thecorrection amounts calculated in step S109. The signal processing unit31 then corrects the various kinds of sensory stimulation information onthe basis of the filter factor.

When the reproduction control is not completed, the signal processingunit 31 performs processing in the order from step S111 to step S105. Instep S105, the signal processing unit 31 performs the reproductioncontrol of the reproducing devices 4 on the basis of the sensorystimulation information that is corrected on the basis of the updatedfilter factor.

Thereafter, the signal processing unit 31 performs the same processing.

During the execution of the processing, the signal processing unit 31terminates the processing of FIG. 9 if reproduced data to be transmittedto the reproducing devices 4 is absent in step S111. The signalprocessing unit 31 may terminate the processing of FIG. 9 by detectingthe end of reproduction by the reproducing devices 4.

In this way, the correction of the sensory stimulation information bythe signal processing unit 31 according to the present embodiment isimplemented.

In the present embodiment, after the completion of the setting of thefilter factor in step S104, the signal processing unit 31 mayrecalculate correction amounts for a delay amount of the output timingand an attenuation of the sensory stimulation information according to achange of the air temperature information in the recording environmentor the reproduction environment.

A change of the air temperature information in the recording environmentmay be a change of video scenes, for example, a change from a video inthe water to a video on the water during reproduction by the videoreproducing device 4 a in the reproduction environment.

A change of the air temperature information in the reproductionenvironment may be, for example, a change of an air temperature in amovie theater or the like due to an increasing number of visitors orswitching of air conditioning.

In this case, when detecting a change of the air temperature informationin the recording environment or the reproduction environment, the signalprocessing unit 31 in step S109 recalculates correction amounts for adelay amount of the output timing and an attenuation of the sensorystimulation information according to the changed air temperatureinformation.

In step S110, the signal processing unit 31 then updates (resets) thefilter factor for correcting a delay amount of the output timing and anattenuation of the sensory stimulation information, on the basis of thecorrection amounts.

In the present embodiment, after the completion of the setting of thefilter factor in step S104, the signal processing unit 31 mayrecalculate correction amounts for a delay amount of the output timingand an attenuation of the sensory stimulation information according to achange of the medium information in the recording environment or thereproduction environment.

A change of the medium information in the recording environment may be achange of video scenes, for example, a change from a video in the waterto a video on the water during reproduction by the video reproducingdevice 4 a in the reproduction environment.

A change of the medium information in the reproduction environment maybe, for example, a dive of the attraction ride of the user Hm into thewater.

In this case, when detecting a change of the medium information in therecording environment or the reproduction environment, the signalprocessing unit 31 in step S109 recalculates correction amounts for adelay amount of the output timing and an attenuation of the sensorystimulation information according to the changed medium information.

In step S110, the signal processing unit 31 then updates (resets) thefilter factor for correcting a delay amount of the output timing and anattenuation of the sensory stimulation information, on the basis of thecorrection amounts.

The present embodiment described an example in which the signalprocessing unit 31 corrects the sensory stimulation information as inFIG. 9 for the video reproducing device 4 a, the sound reproducingdevice 4 b, and the vibration reproducing device 4 c, which serve as thereproducing devices 4. The present embodiment is also applicable to anexample of the processing performed for the video reproducing device 4 aand the sound reproducing device 4 b and an example of the processingperformed for the video reproducing device 4 a and the vibrationreproducing device 4 c. In other words, the sensory stimulationinformation can be corrected by the cooperation of two or more of thereproducing devices 4 according to the present embodiment.

6. Summary

The information processing device (the signal processing unit 31 in thereproduction control device 3) according to the present embodimentincludes: the positional-relationship-information acquisition unit 35that acquires information on the first positional relationship betweenthe recording device 2 and the recording subject, the recording device 2recording the sensory stimulation information in the recordingenvironment for recording the sensory stimulation informationtransmitted from the recording subject, and information on the secondpositional relationship between the recording subject and the user Hm inthe reproduction environment of the recorded sensory stimulationinformation; and the correction unit 36 that corrects the output signalsof the reproducing devices 4 on the basis of the information on thefirst positional relationship and the information on the secondpositional relationship, the reproducing devices 4 reproducing thesensory stimulation information (see FIGS. 7 and 9 ).

The sound signal of the sound reproducing device 4 b serving as thereproducing device 4 and the output signal of the vibration reproducingdevice 4 c serving as the reproducing device 4 are corrected on thebasis of the information on the first positional relationship in therecording environment and the information on the second positionalrelationship in the reproduction environment. This allows the user Hm toperceive sensory stimulation according to the viewing position.

Specifically, the closer to a screen or a display (the display part ofthe video reproducing device 4 a), the higher the level of sounds andvibrations perceivable by the user Hm.

Thus, the user Hm can receive the sensory stimulation information withhigh realism as in the recording environment.

In the present embodiment, if the reproducing devices 4 include multiplekinds of reproducing devices 4 for multiple kinds of sensory stimulationinformation, the information on the second positional relationship isinformation on the relationship between a virtual position (subjectposition T) of the recording subject, which is perceived by the user Hmthrough the display part showing a reproduced image, and the position(perception position A) of the user Hm (see FIGS. 4 and 8 ).

In other words, the output signals of the acoustic information and thevibration information (tactile information) are corrected according tothe video information received by the user Hm.

Thus, the user Hm can be provided with stimuli such as sounds andvibrations according to the direction and the distance of the recordingsubject (e.g., a basketball or a player in the recording of a basketballgame) virtually and visually recognized by the user Hm through a screenor a display (the display part of the video reproducing device 4 a),thereby providing the reproduction environment with realism withoutcausing the user Hm to feel uncomfortable or annoyed.

In the signal processing unit 31 of the present embodiment, thecorrection unit 36 makes the correction according to a change of thefirst positional relationship in the recording environment (e.g., inS106 and S107 of FIG. 9 ).

Thus, the sensory stimulation information is corrected in response to amovement of the recording device 2 or a movement of the recordingsubject in the recording environment.

This can provide the sensory stimulation information with signalstrength that does not cause the user Hm to feel uncomfortable. In otherwords, the user Hm is allowed to perceive various stimuli withoutfeeling uncomfortable or annoyed.

In the present embodiment, a change of the first positional relationshipcorresponds to a change of the position of the recording subject in therecording environment (e.g., in S106 of FIG. 9 ).

Thus, the output signal of the reproducing device 4 is correctedaccording to a change of the position of the recording subject.

In other words, the user Hm is allowed to perceive a change of theposition of the recording subject by changing the various kinds ofsensory stimulation information. Thus, the user Hm is allowed torecognize a change of the position of the recording subject in multipledimensions, thereby increasing realism.

In the present embodiment, a change of the first positional relationshipcorresponds to a change of the position of the recording device 2 in therecording environment (e.g., in S107 of FIG. 9 ).

Thus, the output signal of the reproducing device 4 is correctedaccording to a change of the position of the recording device 2.

In other words, the user Hm is allowed to perceive a change of theposition of the recording device 2, that is, the viewpoint (or visualangle) of the user Hm by changing the various kinds of sensorystimulation information. Thus, the user Hm is allowed to recognize achange of the position of the recording device 2 in multiple dimensions,thereby increasing realism.

In the signal processing unit 31 of the present embodiment, thecorrection unit 36 makes the correction according to a change of thesecond positional relationship in the reproduction environment (e.g., inS108 of FIG. 9 ).

Thus, the output signal of the reproducing device 4 is corrected inresponse to a movement of the reproducing device 4 or a movement of theuser Hm in the reproduction environment.

This can provide the sensory stimulation information with signalstrength that does not cause the user Hm to feel uncomfortable. In otherwords, the user Hm is allowed to perceive various stimuli withoutfeeling uncomfortable or annoyed.

In the present embodiment, a change of the second positionalrelationship corresponds to a change of the position of the user Hm inthe reproduction environment (e.g., in S108 of FIG. 9 ).

Thus, the output signal of the reproducing device 4 is correctedaccording to a change of the perception position A of the user Hm.

In other words, the user Hm is allowed to perceive a change of theposition of the user Hm by changing the various kinds of sensorystimulation information. For example, the sound level of reproductionincreases when the user Hm approaches a screen or a display, whereas thesound level of reproduction decreases when the user Hm moves away from ascreen or the like. Thus, the user Hm can be provided with the sensorystimulation information with high realism as in the recordingenvironment.

In the signal processing unit 31 of the present embodiment, a change ofthe second positional relationship corresponds to a change of theposition of the reproducing device 4 in the reproduction environment(e.g., in S108 of FIG. 9 ). Thus, the output signal of the reproducingdevice 4 is corrected according to a change of the reproducing device 4.

In other words, the user Hm is allowed to perceive a change of theposition of the reproducing device 4 by changing the various kinds ofsensory stimulation information. Thus, the user Hm is allowed torecognize a change of the position of the recording subject in multipledimensions, thereby increasing realism.

In the signal processing unit 31 of the present embodiment, thecorrection unit 36 corrects the output timing of the output signal(e.g., in S103, S105, and S109 of FIG. 9 ).

Hence, for example, in the case of a large distance between the videoreproducing device 4 a (a display part, e.g., a screen), which receivesan outputted video, and the user Hm, the output signal of the soundreproducing device 4 b serving as the reproducing device 4 can beoutputted later.

This can provide the sensory stimulation information for the user Hm ata proper time according to the positional relationship between the userHm and a virtual recording subject, thereby increasing realism.

In the signal processing unit 31 of the present embodiment, thecorrection unit 36 corrects the signal strength of the output signal(e.g., in S103 and S109 of FIG. 9 ). Hence, for example, in the case ofa large distance between the video reproducing device 4 a (a displaypart, e.g., a screen), which receives an outputted video, and the userHm, the output signal of the sound reproducing device 4 b serving as thereproducing device 4 can be reduced in signal strength.

This can provide the user Hm with reproduced sound that is adjusted(corrected) at a proper sound level of reproduction according to thepositional relationship between the user Hm and a virtual recordingsubject, thereby increasing realism.

In the signal processing unit 31 of the present embodiment, thecorrection unit 36 makes the correction according to the environmentinformation in the recording environment (e.g., in S103 of FIG. 9 ).

Thus, the output signal to the reproducing device 4 is corrected suchthat the user Hm actually feels the same sensory stimulation as in therecording environment.

This can provide the sensory stimulation information for the user Hm asif in the recording environment, thereby increasing realism.

In the present embodiment, the environment information is airtemperature information (e.g., in S103 of FIG. 9 ).

Thus, even if a temperature of the environment where the user Hm isactually present is different from a temperature of the recordingenvironment, the output signal to the reproducing device 4 is correctedsuch that the user Hm in the reproduction environment can perceivesensory stimulation to be perceived by the user Hm if the user Hm werein the recording environment.

Thus, the user Hm can be provided with the reproduction environment withhigh realism as in the recording environment.

In the present embodiment, the environment information is mediuminformation (e.g., in S103 of FIG. 9 ).

Thus, even if the medium (e.g., water or air) of the reproductionenvironment is different from the medium of the recording environment,the output signal to the reproducing device 4 is corrected such that theuser Hm in the reproduction environment can perceive sensory stimulationto be perceived by the user Hm if the user Hm were in the recordingenvironment.

This can provide the user Hm with the reproduction environment with highrealism as in the recording environment.

In the signal processing unit 31 of the present embodiment, thecorrection unit 36 corrects the output signal of the reproducing device4 that reproduces the acoustic information as the sensory stimulationinformation (e.g., in S103 and S109 of FIG. 9 ).

Thus, the output signal to the reproducing device 4 is corrected for theacoustic information that tends to arrive at different times when therecording position and the reproduction position are different from eachother.

This can output the acoustic information from the reproducing device 4without causing the user Hm to feel uncomfortable.

In the signal processing unit 31 of the present embodiment, thecorrection unit 36 corrects the output signal of the reproducing device4 that reproduces the vibration information as the sensory stimulationinformation (e.g., in S103 and S109 of FIG. 9 ).

Thus, the output signal to the reproducing device 4 is corrected for thevibration information that tends to arrive at different times when therecording position and the reproduction position are different from eachother.

This can output the acoustic information from the reproducing device 4without causing the user Hm to feel uncomfortable.

The recording/reproduction system according to the present embodimentincludes: the recording device 2 including asensory-stimulation-information acquisition unit that acquires thesensory stimulation information transmitted from the recording subject,and a detection unit that detects information on the first positionalrelationship between the recording device 2 (recording position R) andthe recording subject (subject position T); and the reproduction controldevice 3 including the positional-relationship-information acquisitionunit 35 that acquires the information on the first positionalrelationship and information on the second positional relationshipbetween the recording subject (subject position T) and the user Hm(perception position) in the reproduction environment of the recordedsensory stimulation information, and the correction unit 36 thatcorrects the output signals of the reproducing devices for thereproduction of the sensory stimulation information on the basis of theinformation on the first positional relationship and the information onthe second positional relationship (e.g., FIGS. 1 and 9 ).

The recording/reproduction system configured as an embodiment can obtainthe same operation and effects as in the foregoing embodiment.

An information processing method according to the present embodiment isan information processing method in which the information processingdevice is caused to perform: positional relationship informationacquisition to acquire the information on the first positionalrelationship between the recording device 2 (recording position R) andthe recording subject (subject position T), the recording device 2performing recording in the recording environment for recording thesensory stimulation information transmitted from the recording subject,and the information on the second positional relationship between therecording subject (subject position T) and the user Hm (perceptionposition) in the reproduction environment of the recorded sensorystimulation information, and a correction to correct the output signalof the reproducing device 4 for the reproduction of the sensorystimulation information on the basis of the information on the firstpositional relationship and the information on the second positionalrelationship. The information processing method as an embodiment canobtain the same operation and effects as the information processingdevice according to the foregoing embodiment.

The functions of the information processing device can be implemented assoftware processing by, for example, a CPU or a DSP. The softwareprocessing is executed based on a program.

A program according to the present embodiment is a program that causesthe information processing device to perform: apositional-relationship-information acquisition function of acquiringthe information on the first positional relationship between therecording device and the recording subject, the recording deviceperforming recording in the recording environment for recording thesensory stimulation information transmitted from the recording subject,and the information on the second positional relationship between therecording subject and the user in the reproduction environment of therecorded sensory stimulation information, and

a correction function of correcting the output signals of thereproducing devices for the reproduction of the sensory stimulationinformation on the basis of the information on the first positionalrelationship and the information on the second positional relationship.

With this program, the information processing device according to theforegoing embodiment can be implemented as the reproduction controldevice 3. In the present embodiment, the reproduction control device 3performs the processing of FIG. 9 on the basis of the program, therebyimplementing the present technique. The present technique can be alsoimplemented by processing performed by the recording device 2 and thereproducing devices 4 according to the program recorded in the recordingdevice 2 and the reproducing devices 4. Alternatively, the processingmay be implemented by executing the program in an external server.

The program can be recorded in advance in a recording medium embedded ina device such as a computer device or a ROM or the like in amicrocomputer including a CPU.

Alternatively, the program can be temporarily or permanently stored(recorded) on a removable recording medium such as a flexible disc, aCD-ROM (Compact Disc Read Only Memory), an MO (Magneto optical) disc, aDVD, a Blu-ray Disc (registered trademark), a magnetic disk, asemiconductor memory, or a memory card. The removable recording mediumcan be provided as so-called package software.

The program can be installed from the removable recording medium to apersonal computer and can also be downloaded from a download site via anetwork such as the Internet or a local area network (LAN).

The program is suitable for widely providing the reproduction controldevice 3 according to the embodiment. For example, the program isdownloaded to a personal computer, a portable information processingdevice, a mobile phone, a game console, or an AV (Audio Visual) device,causing the personal computer or the like to function as thereproduction control device 3.

7. Present Technique

The present technique can be also configured as follows:

(1)

An information processing device including: apositional-relationship-information acquisition unit that acquiresinformation on a first positional relationship between a recordingdevice and a recording subject, the recording device performingrecording in a recording environment for recording sensory stimulationinformation transmitted from the recording subject, and information on asecond positional relationship between the recording subject and a userin a reproduction environment of the recorded sensory stimulationinformation, and a correction unit that corrects the output signals ofreproducing devices for the reproduction of the sensory stimulationinformation on the basis of the information on the first positionalrelationship and the information on the second positional relationship.

(2)

The information processing device according to (1), wherein if thereproducing devices include multiple kinds of reproducing devices formultiple kinds of sensory stimulation information, the information onthe second positional relationship is information on the relationshipbetween a virtual position of the recording subject, which is perceivedby the user through the display part showing a reproduced image, and theposition of the user.

(3)

The information processing device according to (1) or (2), wherein thecorrection unit makes the correction according to a change of the firstpositional relationship in the recording environment.

(4)

The information processing device according to (3), wherein the firstpositional relationship is changed according to a change of the positionof the recording subject in the recording environment.

(5)

The information processing device according to (3) or (4), wherein thefirst positional relationship is changed according to a change of theposition of the recording device in the recording environment.

(6)

The information processing device according to any one of (1) to (5),wherein the correction unit makes the correction according to a changeof the second positional relationship in the reproduction environment.

(7)

The information processing device according to (6), wherein the secondpositional relationship is changed according to a change of the positionof the user in the reproduction environment.

(8)

The information processing device according to (6) or (7), wherein thesecond positional relationship is changed according to a change of theposition of the reproducing device in the reproduction environment.

(9)

The information processing device according to any one of (1) to (8),wherein the correction unit corrects the output timing of the outputsignal.

(10)

The information processing device according to any one of (1) to (9),wherein the correction unit corrects the signal strength of the outputsignal.

(11)

The information processing device according to any one of (1) to (10),wherein the correction unit makes the correction on the basis of adifference between environment information in the recording environmentand environment information in the reproduction environment.

(12)

The information processing device according to (11), wherein thedifference of the environment information is a difference in airtemperature.

(13)

The information processing device according to (11), wherein thedifference of the environment information is a difference in medium.

(14)

The information processing device according to any one of (1) to (13),wherein the correction unit corrects the output signal of thereproducing device that reproduces acoustic information as the sensorystimulation information.

(15)

The information processing device according to any one of (1) to (14),wherein the correction unit corrects the output signal of thereproducing device that reproduces vibration information as the sensorystimulation information.

(16)

An information processing method in which an information processingdevice is caused to perform: positional relationship informationacquisition to acquire information on a first positional relationshipbetween a recording device and a recording subject, the recording deviceperforming recording in a recording environment for recording sensorystimulation information transmitted from the recording subject, andinformation on a second positional relationship between the recordingsubject and a user in a reproduction environment of the recorded sensorystimulation information, and

a correction to correct the output signals of reproducing devices forthe reproduction of the sensory stimulation information on the basis ofthe information on the first positional relationship and the informationon the second positional relationship.(17)

A program that causes an information processing device to perform: apositional-relationship-information acquisition function of acquiringinformation on a first positional relationship between a recordingdevice and a recording subject, the recording device performingrecording in a recording environment for recording sensory stimulationinformation transmitted from the recording subject, and information on asecond positional relationship between the recording subject and a userin the reproduction environment of the recorded sensory stimulationinformation, and

a correction function of correcting the output signals of thereproducing devices for the reproduction of the sensory stimulationinformation on the basis of the information on the first positionalrelationship and the information on the second positional relationship.(18)

A recording/reproduction system including a recording device and areproduction control device,

wherein the recording device includes:a sensory-stimulation-information acquisition unit that acquires sensorystimulation information transmitted from a recording subject; anda detection unit that detects information on a first positionalrelationship between the recording device and the recording subject, andthe reproduction control device includes:a positional-relationship-information acquisition unit that acquires theinformation on the first positional relationship and information on asecond positional relationship between the recording subject and a userin the reproduction environment of the recorded sensory stimulationinformation, anda correction unit that corrects the output signals of reproducingdevices for the reproduction of the sensory stimulation information onthe basis of the information on the first positional relationship andthe information on the second positional relationship.

Finally, the advantageous effects described in the present disclosureare exemplary and not limited, and may have other advantageous effectsor may have some of the advantageous effects described in the presentdisclosure.

The embodiment described in the present disclosure is merely exemplary,and the present technique is not limited to the foregoing embodiment.Therefore, it goes without saying that various changes aside from theforegoing embodiment can be made according to the design and the likewithout departing from the technical spirit of the present technique. Itshould be noted that all the combinations of the configurationsdescribed in the embodiment are not always essential for solving theproblem.

Reference Signs List

1 Recording/reproduction system

2 Recording device

3 Reproduction control device

4 Reproducing device

4 a Video reproducing device

4 b Sound reproducing device

4 c Vibration reproducing device

10 Camera unit

11 Image signal processing unit

12 Sound input unit

13 Sound signal processing unit

14 Vibration detecting unit

15 Vibration signal processing unit

16 Sensor unit

17 Encoding unit

30 Decoding unit

31 Signal processing unit

35 Positional-relationship information acquisition unit

36 Correction unit

T Subject position

R Recording position

A, B Perception position

Hm User

1. An information processing device comprising: apositional-relationship-information acquisition unit that acquiresinformation on a first positional relationship between a recordingdevice and a recording subject, the recording device performingrecording in a recording environment for recording sensory stimulationinformation transmitted from the recording subject, and information on asecond positional relationship between the recording subject and a userin a reproduction environment of the recorded sensory stimulationinformation, and a correction unit that corrects output signals ofreproducing devices for reproduction of the sensory stimulationinformation on a basis of the information on the first positionalrelationship and the information on the second positional relationship.2. The information processing device according to claim 1, wherein ifthe reproducing devices include multiple kinds of reproducing devicesfor multiple kinds of sensory stimulation information, the informationon the second positional relationship is information on a relationshipbetween a virtual position of the recording subject, which is perceivedby the user through a display part showing a reproduced image, and aposition of the user.
 3. The information processing device according toclaim 1, wherein the correction unit makes the correction according to achange of the first positional relationship in the recordingenvironment.
 4. The information processing device according to claim 3,wherein the first positional relationship is changed according to achange of a position of the recording subject in the recordingenvironment.
 5. The information processing device according to claim 3,wherein the first positional relationship is changed according to achange of a position of the recording device in the recordingenvironment.
 6. The information processing device according to claim 1,wherein the correction unit makes the correction according to a changeof the second positional relationship in the reproduction environment.7. The information processing device according to claim 6, wherein thesecond positional relationship is changed according to a change of aposition of the user in the reproduction environment.
 8. The informationprocessing device according to 6, wherein the second positionalrelationship is changed according to a change of a position of thereproducing device in the reproduction environment.
 9. The informationprocessing device according to claim 1, wherein the correction unitcorrects an output timing of the output signal.
 10. The informationprocessing device according to claim 1, wherein the correction unitcorrects signal strength of the output signal.
 11. The informationprocessing device according to claim 1, wherein the correction unitmakes the correction according to environment information in therecording environment.
 12. The information processing device accordingto claim 11, wherein the environment information is air temperatureinformation.
 13. The information processing device according to claim11, wherein the environment information is medium information.
 14. Theinformation processing device according to claim 1, wherein thecorrection unit corrects the output signal of the reproducing devicethat reproduces acoustic information as the sensory stimulationinformation.
 15. The information processing device according to claim 1,wherein the correction unit corrects the output signal of thereproducing device that reproduces vibration information as the sensorystimulation information.
 16. An information processing method in whichan information processing device is caused to perform: positionalrelationship information acquisition to acquire information on a firstpositional relationship between a recording device and a recordingsubject, the recording device performing recording in a recordingenvironment for recording sensory stimulation information transmittedfrom the recording subject, and information on a second positionalrelationship between the recording subject and a user in a reproductionenvironment of the recorded sensory stimulation information; and acorrection to correct output signals of reproducing devices forreproduction of the sensory stimulation information on a basis of theinformation on the first positional relationship and the information onthe second positional relationship.
 17. A program that causes aninformation processing device to perform: apositional-relationship-information acquisition function of acquiringinformation on a first positional relationship between a recordingdevice and a recording subject, the recording device performingrecording in a recording environment for recording sensory stimulationinformation transmitted from the recording subject, and information on asecond positional relationship between the recording subject and a userin a reproduction environment of the recorded sensory stimulationinformation; and a correction function of correcting output signals ofreproducing devices for reproduction of the sensory stimulationinformation on a basis of the information on the first positionalrelationship and the information on the second positional relationship.18. A recording/reproduction system comprising a recording device and areproduction control device, wherein the recording device includes: asensory-stimulation-information acquisition unit that acquires sensorystimulation information transmitted from a recording subject; and adetection unit that detects information on a first positionalrelationship between the recording device and the recording subject, andthe reproduction control device includes: apositional-relationship-information acquisition unit that acquires theinformation on the first positional relationship and information on asecond positional relationship between the recording subject and a userin a reproduction environment of the acquired sensory stimulationinformation; and a correction unit that corrects output signals ofreproducing devices for reproduction of the sensory stimulationinformation on a basis of the information on the first positionalrelationship and the information on the second positional relationship.